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Executive Summary 



This document presents the report on the formative evaluation of the Aboriginal Skills 
and Employment Partnership (ASEP) program conducted by Human Resources and 
Skills Development Canada (HRSDC). 



Background 

The five-year, $85M ASEP initiative is a targeted Aboriginal s ki lls development program 
designed to promote maximum employment for Aboriginal people on major economic 
developments through a collaborative partnership approach, leading to lasting benefits for 
Aboriginal individuals, communities and families. It is designed to address a broad spectrum 
of skills and learning needs and provide access to jobs. 

The ASEP program has funded nine diverse projects in resource-based sectors, including 
mining, forestry, oil and gas, hydro development, fishery, and construction. Each project is 
located in a different community or group of communities with varying project scopes. 
Funding levels range from approximately $3M to $22M. The nine projects are located across 
the country in rural and urban areas, as well as northern and southern settings. 

The immediate objectives of the ASEP program are to: 

• Foster collaboration, partnerships, alliances and networks between government, the private 
sector, non-government agencies and Aboriginal groups to maximize the employment 
opportunities available to Aboriginal people in areas of major economic development by 
leveraging investment of resources, in-kind and financial, from ASEP project partners; 

• Build the capacity of communities to address human resources development needs related 
to economic opportunities; and 

• Increase the quantity of employment-related skills training available to Aboriginal people 
in areas of major economic development beyond the level possible by reliance solely on 
Aboriginal Human Resources Development Agreement (AHRDA) funding. 

The long term objectives of the ASEP program are to: 

• Increase the number of Aboriginal people employed in the Canadian labour market; 

• Reduce the barriers to employment for Aboriginal people such as low educational 
attainment and lack of opportunity; and 

• Address a broad spectrum of skills and learning needs and provide access to jobs. 
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Evaluation Scope and Methods 

The Results-based Management and Accountability Framework developed for the ASEP 
program guides the continuous monitoring and evaluation process. The goal of the process is to 
determine the success of the projects in terms of implementation and effectiveness. The process 
is comprised of multiple components to be implemented at various stages of the program. 

The overall objectives of the formative evaluation component were to assess the rationale, 
relevance, implementation, results to-date, and accountability of the ASEP program. 

The methodology developed and implemented for the formative evaluation included: 

• Five case study projects which included key informant interviews, discussion groups, 
a project document review, and socio-economic profiles; and 

• Assessment and analysis of client case management and administrative data for all nine 
ASEP projects. 



Key Findings and Conclusions 
Program Rationale and Relevance 

• There is a strong continued need for the ASEP program, as demonstrated through the 
case studies and the socio-economic profiles. The ASEP program is viewed as 
addressing needs of both the Aboriginal communities and industry. Specifically, ASEP 
interventions are perceived as relevant to the needs identified within Aboriginal 
communities to develop skills and gain work experience in order to fully participate in 
key industrial projects in their regions. 

• The ASEP program continues to reflect the broad federal government Aboriginal 
policy directions as outlined in recent planning and policy documents. 



Program Implementation 

• Overall, most activities under the various ASEP projects have been implemented as 
originally planned. Information collected from project documents, partners and staff 
demonstrated that key assessment and training activities had been undertaken and most 
planned components were underway at the time of the interviews. 

• Overall, there are significant gaps in the case management and financial data required 
for evaluation and accountability. In all but one of the nine projects, remedial work is 
needed to ensure that data accurately reflect project activity. The most serious gaps 
with respect to case management data are: 

- There is no electronic case management data available for a significant proportion 
of clients. As a result, complete case management data are neither readily available 
on-site, nor available from HRSDC’s Standard Data File (SDF). 
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- Data stored in case management systems are miscoded in the majority of cases. 

• The financial data, for most projects, is more complete; however, for all projects, work is 
required to link project costs to individual clients. 

• Overall, the services were perceived as appropriate for the target clients. In many instances, 
training and work placements had been tailored to meet the specific needs of clients, and 
interventions had been adjusted based on evaluations and feedback received from 
participants. Even with the tailoring and adjustments, there was the acknowledgement that 
the ASEP projects could not accommodate some issues such as providing services for 
those who had very low education levels who requested training, and some contextual 
issues such as participants having to leave the community to find work until the large 
projects are underway. 

• The actual expenditures, as reported in annual reports and summary financial reporting, 
were less than planned expenditures for the initial stages of the projects examined. This is 
to be expected given the delays encountered in implementing activities during the first 6 to 
12 months with many projects, and the nature of the activities during the start-up period. 

• There was considerable variability in the amount of effort and resources the various ASEP 
projects had invested to date in developing the capacity of governing structures, ASEP staff 
and delivery agents. While two of the case study projects reported few activities in these 
areas beyond developing the basic by-laws and procedures associated with becoming an 
incorporated organization, two of the case study projects indicated that considerable effort 
had been made in developing capacity of delivery agents, ASEP staff and Boards 
of Directors. 

• The ASEP projects have developed and implemented marketing, promotional and 
communication activities to ensure that targeted communities are reached. Activities included 
direct outreach via partners’ networks, community meetings, and direct promotional 
techniques such as flyers, newsletters, pamphlets and posters. The case studies identified 
challenges to ensuring that targeted communities are reached, with the main challenge 
identified as the relatively rapid turnover of staff within the various communities, which 
requires ongoing promotional and training activities to keep them adequately informed. 

• The quality of the coordination between the ASEP structures, delivery agents and other 
employment services for Aboriginal people seemed to vary considerably between 
projects. Where the coordination is perceived as being successful, the key factors 
considered to contribute to this are a solid partnership with a shared vision, 
communication between partners and ASEP staff, solid case management systems, 
development and adoption of common tools and templates, and ongoing identification 
of common issues and areas of concerns. Where coordination has been perceived as 
less successful, issues identified included large staff turnover, overlapping services, 
and perceived competition between organizations. 
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Program Results To-Date 

• In most cases, the ASEP projects were identified as increasing the ability of organizations to 
address the needs of Aboriginal populations. In a few cases, the projects were perceived to 
have li mited success in this area by providing services that overlap with existing 
opportunities provided by already established programming. 

• There were a few examples provided during the evaluation that demonstrated how new 
knowledge generated from ASEP projects was beginning to be transferred and potentially 
incorporated into other related services. 

• Based on preliminary results, the ASEP projects are demonstrating both tangible and 
intangible outcomes at the individual level. The projects have met most of their early results 
targets, and individuals are reported to have enhanced their skills and been provided with 
work experience opportunities. As the projects progress, the challenge faced by some 
projects will be to assist clients in returning to employment given some of the delays in the 
timelines for the associated large projects, and sudden downturns in some sectors. 



Accountability 

• Developing adequate case management and financial systems to ensure that there is no 
duplication or displacement of non-federal investments has been challenging for many of 
the projects, as identified in the data assessments. Although the development of case 
management and financial systems has been challenging, the active involvement of partners 
under a partnership model can assist in minimizing duplication and overlap of investments. 
The actual measurement of incrementality of investments was not possible in this formative 
evaluation given the lack of baseline investment data prior to the implementation of the 
ASEP program. 
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Management Response 



The Aboriginal Affairs Directorate (AAD) of the Skills and Employment Branch (SEB), 
Human Resources and Skills Development Canada (HRSDC) would like to thank all those 
who participated in formulating and conducting this evaluation of the Aboriginal Skills and 
Employment Partnership (ASEP) program. 

In particular, we acknowledge the contributions of the nine ASEP projects across the country: 
ASEP-NB, Inc.; James Bay Employment and Training; Wuskwatim and Keeyask Training 
Consortium; Trade Winds to Success Training Society; Wood Buffalo Partners in Aboriginal 
Training Association; VanASEP Training Society; Aboriginal Futures Society; Mine 
Training Society; and, the Nunavut Fisheries Training Consortium. 

We acknowledge the observations provided and note these findings. We have taken action to 
strengthen the implementation and accountability of the ASEP program based on this 
evaluation in concert with other lines of evidence. 

ASEP contribution agreements enable partnerships between Aboriginal organizations, major 
employers and provinces/territories to deliver programs and services that ensure Aboriginal 
people have the skills necessary for long term jobs. The overall objective of the ASEP 
program is sustainable jobs for Aboriginal people on major economic developments leading 
to lasting benefits for Aboriginal people, communities and families. 

This formative review of ASEP was conducted as a result of the commitment to review the 
program midstream to identify best practices, as well as identify issues that need to be 
addressed to make the program more effective. 

The program was expanded by $105M in 2007 and extended to 2012 and the findings are 
helping inform the implementation of the new ASEP projects starting in fiscal year 
2008/2009. 

The goal of the evaluation is to determine the success of the projects in terms of 
implementation and effectiveness. The objectives of the ASEP formative evaluation were to 
assess the rationale, relevance, implementation success to date, and accountability of the 
ASEP program. The key findings and conclusions of the formative evaluation of the ASEP 
program are outlined in terms of Program Rationale and Relevance; Program 
Implementation; Program Success; and Accountability. 

The following outlines AAD’s Management Response and commitments to the evaluation 
presented by the HRSDC Evaluation Directorate. 



Program Rationale and Relevance 

Key Finding 1: There is a strong continued need for the ASEP program, as demonstrated 
through the case studies and the socioeconomic profiles. The ASEP program is viewed as 
addressing needs of both the Aboriginal communities and industry. Specifically, ASEP 
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interventions are perceived as relevant to the needs identified within Aboriginal communities 
to develop skills and gain work experience in order to fully participate in key industrial 
projects in their regions. 

ASEP is a unique federal government initiative, in that it brings together the supply and 
demand sides of the labour market, helping to ensure that Aboriginal people have the s ki lls 
required for long term jobs in and near their communities that are tied directly to major 
economic developments. The ASEP interventions are perceived as relevant to Aboriginal 
communities and private sector/industry as training within the ASEP training to employment 
plan is tied directly to long term jobs. This key finding is further supported by the fact that 
around 60 proposals were received for funding as a result of the ASEP Request for Proposals 
that closed September 7, 2007, requesting approximately $300M indicating a high interest in 
the program. 

Key Finding 2: The ASEP program continues to reflect the broad federal government 
Aboriginal policy directions as outlined in recent planning and policy documents. 

This formative evaluation references the HRSDC 2006-2007 Report on Plans and Priorities; 
the 2006 Speech from the Throne; Canada’s Innovation Strategy and the Social Union 
Framework Agreement as providing policy relevance for the ASEP program. The ASEP 
program is also consistent with the government’s commitments in Advantage Canada (the 
November 2006 Economic Update), Budget 2007, the October 2007 Speech from the Throne 
as well as Budget 2008. Budget 2008 states: The Government has made significant progress 
on Aboriginal issues over the past two years, working with willing partners to achieve 
tangible results. This new, practical approach of working with Aboriginal governments and 
organizations, provinces and territories, and the private sector to address clear priorities in 
an effective and targeted manner is paying off and producing results. 



Program Implementation 

Key Finding 3: Ch’erall, most activities under the various ASEP projects have been 
implemented as originally planned. Information collected from project documents, partners 
and staff demonstrated that key assessment and training activities had been undertaken and 
most planned components were underway at the time of the inten’iews. 

Key Finding 4: Overall, there are significant gaps in the case management and financial data 
required for evaluation and accountability. In cdl but one of the nine projects, remedial work 
is needed to ensure that data accurately reflect project activity. Tire most serious gaps with 
respect to case management data are: 

• There is no electronic case management data available for a significant proportion of 
clients. As a result, complete case management data are neither readily available on-site, 
nor available from HRSDC Standard Data File (SDF). 

• Data stored in case management systems are miscoded in the majority of training cases. 
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The financial data, for most projects, is more complete; however, for cdl projects, work is 
required to link project costs to individual clients. 

Case management and financial data within the ASEP projects are instrumental in determining 
how the projects are being implemented, the progression on their plan, the success of the 
project and the ASEP program overall. Because of a delay regarding approval to collect the 
Social Insurance Number (SIN), the ASEP projects were behind in capturing client 
information in a case management system and uploading data to HRSDC SDF. AAD 
continues to work with the ASEP projects on a priority basis to address the gaps in the client 
case management and financial data of each project so that issues identified in the key findings 
are addressed. 

AAD also took action to recruit staff with in-depth knowledge of case management and 
financial systems, to provide the support and assistance to ASEP projects in capturing all 
client data accurately as well as linking the financial data to the clients. Two Senior Program 
Officer positions within AAD are responsible for providing support, training and monitoring 
of the case management data for existing projects as well as for new ASEP projects. Support 
will be provided to new projects in selecting and/or developing an appropriate case 
management system to collect the data required, as well as linking the financial data to 
individual clients. In-depth support continues to be provided to the ASEP projects, both 
existing and new, to closely monitor the case management and financial data systems to help 
min im ize gaps in the data for evaluation and accountability purposes. 

As a result of the directive on Specialization and Concentration and the recent Separation of 
Duties audit, changes have been made within the structure of SEB which have resulted in the 
creation of the Delivery Support Division (DSD). The DSD is a common back office for all 
grants and contributions programs implemented by S ki lls and Employment Branch (SEB). 
Oversight and monitoring is now also the responsibility of the DSD. Claim verification and 
financial monitoring will be completed by the DSD in conjunction with AAD staff. The role 
of AAD staff in project management will be to help ensure the ASEP project activities are 
being implemented as planned, providing support as required and closely monitoring the case 
management data and data quality. 

Key Finding 5: Overall, the services were perceived as appropriate for the target clients. 
In many instances, training and work placements had been tailored to meet the specific needs 
of clients, and interx’entions had been adjusted based on evaluations and feedback received 
from participants. Even with the tailoring and adjustments, there was the acknowledgement 
that the ASEP projects could not accommodate some issues such as providing sendees for 
those who had very low education levels who requested training, and some contextual issues 
such as participants having to leave the community to find work until the large projects 
are underway. 

ASEP was designed to complement the Aboriginal Human Resources Development Strategy 
(AHRDS) by providing targeted training and supports to increase Aboriginal participation on 
major economic developments. The demands of these major projects are beyond the ability for 
the AHRDS agreement holders on their own, to respond. ASEP provides the ability to lever 
funding from partners with an interest in the major economic opportunity. The formalized 
partnerships include, at a minimum, representatives from the Aboriginal organizations in the 
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area, and the employer(s) involved in the project as well as provincial/territorial governments 
and others as needed. 

ASEP projects require significant front end development to ensure that the partnership is well 
grounded. Because the projects are five or less years, it is recognized that there may be some 
individuals (particularly those with multiple barriers) that will not have time to obtain the s ki lls 
necessary within the time frame of the Aboriginal training to employment plan in order to 
secure high s ki lled long term jobs on the major economic development. AAD will ensure that 
information regarding essential s ki lls is shared with each project and encourage ASEP projects 
to implement essential skill profiles for the long term jobs, and provide interventions based on 
the essential s ki ll profile. 

AAD will continue to work with the partners to help identify referral opportunities within the 
community, including to the AHRDA, to assist individuals to obtain the skills they are 
interested in pursuing as well as to provide support to people who may have to leave their 
community for periods of time to obtain necessary work experience. While the goal is to 
provide training and work experience for individuals in or near their communities, instances 
exist where this may not be feasible due to delay in start up of the project. In these instances, 
HRSDC will continue to work with the project partners including employers to identify 
appropriate opportunities to address issues that arise in the Aboriginal training to employment 
plan. 

Key Finding 6: The actual expenditures, as reported in annual reports and summary 
financial reports, were less than planned expenditures for the initial stages of the projects 
examined. Tins is to be expected given the delays encountered in implementing activities 
during the first 6 to 12 months with many projects, and the nature of the activities during the 
start-up period. 

The initial $85M investment in ASEP was announced in October 2003 and the first ASEP 
contribution agreement supported from these funds was signed in August 2004; the last (ninth 
agreement) was signed in February 2006. Each ASEP project is unique. A lesson learned from 
the initial set of projects was that implementation of project activities and actual expenditures 
required more time than first anticipated. As AAD enters into new ASEP contribution 
agreements with partnerships this knowledge will help establish realistic timelines around 
start-up timeframes and implementation of project activities. A minimum six month 
implementation phase and start up-phase will be provided to establish the project, the 
partnership and the appropriate mechanisms to implement and manage the ASEP project. 

Key Finding 7: There was considerable variability in the amount of effort and resources 
various ASEP projects had invested to date in developing the capacity of governing structures, 
ASEP staff and delivery agents. While two of the case study projects reported few activities in 
these areas beyond developing the basic by-laws and procedures associated with becoming an 
incorporated organization, two of the case study projects indicated that considerable effort 
had been made in developing capacity of delivery agents, ASEP staff and Board of Directors. 

This finding from the first nine ASEP projects co nfir ms that the amount of time and effort given 
to developing capacity of the governing structure, staff and delivery agents varies from project to 
project. The implementation of the existing ASEP projects demonstrated that the capacity of the 
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governing structure and staff is instrumental to the successful implementation of the ASEP 
project. Within the first months of the project start up it is extremely important to develop the 
capacity of the partners and an understanding of the roles and responsibilities of the incorporated 
partnership and its governing structure. It is also important to follow the project start-up with 
continuous activities within the first six months to ensure the partners, governing body, staff and 
delivery agents fully understand their respective roles, responsibilities and the purpose of 
the project. 

AAD has developed and compiled numerous tools and documents to assist new partnerships in 
the implementation of their project, such as information on incorporation; governance manuals 
from successful partnerships as well as contact information regarding board development 
workshops/seminars; sample job descriptions, employment contracts, service contracts, sub- 
agreements, request for proposals, and the like. These tools have been compiled by project phase 
and the tools are being shared with proponents via compact disc so they have access to the files 
and may uti li ze and alter them to fit their needs. HRSDC AAD Project Officers will ensure that 
the information is shared appropriately and will work with the ASEP proponents to ensure that 
capacity building of the partnership, board, staff and delivery agents is a focus at the start of the 
contribution agreement with an appropriate plan in place for each project. AAD will also explore 
posting appropriate tools, forms and documentation on its website, so that information is readily 
available to all project proponents. 

Key Finding 8: The ASEP projects have developed and implemented marketing, promotional 
and communication activities to ensure that targeted communities are reached Activities 
included direct outreach via partners ’ networks, community meetings, and direct promotional 
techniques such as flyers, newsletters, pamphlets and posters. The case studies identified 
challenges to ensuring that targeted communities are reached, with the main challenge 
identified as the relatively rapid turnover of staff within the various communities, which 
requires ongoing promotional and training activities to keep them adequately informed. 

It is common for First Nation communities to experience turnover of administrative personnel 
which can sometimes create challenges with respect to communications. It is acknowledged 
that communication, marketing and promotional activities must be ongoing in order to keep 
communities informed of the project activities and to reach the targeted population. AAD 
works with the partnerships including Aboriginal organizations, provincial and territorial 
governments, and industry to identify resources they may have to promote the projects and to 
ensure good communication regarding project activities. ASEP projects have been encouraged 
to establish websites as a means of communication to keep stakeholders, communities and 
individuals up to date on the project. 

Key Finding 9: The quality of the coordination between the ASEP structures, delivery agents 
and other employment services for Aboriginal people seemed to vary considerably between 
projects. Where the coordination is perceived as being successful, the key factors considered 
to contribute to this are a solid partnership with a shared vision, communication between 
partners and ASEP staff, solid case management systems, development and adoption of 
common tools and templates, and ongoing identification of common issues and areas of 
concerns. Where coordination has been perceived as less successful, issues identified included 
large staff turnover, overlapping services, and perceived competition between organizations. 
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This key finding is tied closely to the capacity of the partnership, its governing structure, staff 
and delivery agents as outlined above. Common understanding is instrumental in ensuring the 
effectiveness of the partnership as is an understanding of the programs and services of the 
different partners in order to help reduce overlapping programs and services. ASEP initiatives 
are targeted and tailored to the opportunity and labour market demand. They complement 
existing programs and services, not replace them. A complete understanding of all partners’ 
programs and services and how they relate and complement rather than compete with each 
other is an essential piece of the partnership’s capacity building. AAD works closely with the 
partnerships during the establishment of their governing structure to lay the ground work for a 
strong partnership and shared vision. Linkages have been made, and will continue to be made, 
with existing ASEP projects that have demonstrated strong coordination with the partnership. 



Program Success 

Key Finding 10: In most cases, the ASEP projects were identified as increasing the ability of 
organizations to address the needs of Aboriginal populations. In a few cases, the projects were 
perceived to have limited success in this area by providing services that overlap with existing 
opportunities provided by already established programming. 

As identified in the previous key finding, in order to mitigate the overlap with existing 
opportunities, AAD works with partners to help ensure a complete understanding of the 
programs and services available in the targeted communities and how the ASEP projects 
complement and build on existing programs and services in the communities. In the recent 
ASEP RFP, the partnership was required to identify existing programs and services in the 
community and how the proposed ASEP programs and services would complement these 
existing services. AAD will work with the partnerships to identify AHRDA, Service Canada, 
Indian and Northern Affairs Canada (ENAC) and Provincial-Territorial (PT) programs that 
may exist in order to ensure all entities are working together to address the needs of the 
Aboriginal population. 

Key Finding 11: There were a few examples provided during the evaluation that 
demonstrated how new knowledge generated from ASEP projects was beginning to be 
transferred and potentially incorporated into other related services. 

AAD has developed a comprehensive communication strategy that incorporates the 
identification of best practices within the ASEP projects, and ensures successes are 
communicated via newsletters, website updates, news articles, presentations at national and 
international forums, and so forth. AAD also holds annual sessions inviting all the partnerships 
of the current projects to share and communicate successes and best practices. 

Key Finding 12: Based on preliminary results, the ASEP projects are demonstrating both 
tangible and intangible outcomes at the individual level. The projects have met most of their 
early results targets, and individuals are reported to have enhanced their skills and been 
provided with work experience opportunities. As the projects progress, the challenge faced by 
some projects will be to assist clients in returning to employment given some of the delays in 
the timelines for the associated large projects, and sudden downturns in some sectors. 
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AAD continues to provide support to ASEP projects in the capture of client data to ensure 
their case management system and the HRSDC SDF accurately reflect the outcomes of their 
project and the activities undertaken. In addition, AAD agrees that a downturn in an industry 
or delays in the timelines of the major economic project can affect the outcomes of the ASEP 
project. AAD closely monitors the projects and works with the partnerships to identify other 
private sector partners and/or opportunities that exist in order to meet the targets established in 
the contribution agreement, or alter the timelines on the Aboriginal training to employment 
plan if appropriate. 



Accountability 

Key Finding 13: Developing adequate case management and financial systems to ensure 
that there is no duplication or displacement of non-federal investments has been challenging 
for many of the projects, as identified in the data assessments. Although the development of 
case management and financial systems has been challenging, the active involvement of the 
partners under a partnership model can assist in minimizing duplication and overlap of 
investments. The actual measurement of incrementality of investments was not possible in 
this formative evaluation given the lack of baseline data prior to the implementation of the 
ASEP program. 

As a result of the data assessments conducted on the nine existing ASEP projects, AAD 
worked with each project to develop and implement an action plan to address identified data 
gaps. As ASEP projects involve multiple partners, and often separate but duplicate reporting 
structures, AAD will work with all funders and partners on streamlining reporting 
requirements, and the associated systems, so that one report may serve the need for all. 
Identification of data systems that may serve the needs of multiple funders, without requiring 
two separate data systems, will be explored with the partners in order to minimize duplication 
and overlap. AAD will work closely with the partners, Service Canada, the AHRDAs and the 
provinces/territories to establish adequate case management and financial systems from the 
start up of new projects and provide partnerships with the support and training required to 
adequately capture the data and report on investments. 
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1. Introduction 



This document presents the report on the formative evaluation of the Aboriginal Skills and 
Employment Partnership (ASEP) program conducted by Human Resources and S ki lls 
Development Canada (HRSDC), covering the period from November 2003 to November 
2006. The report is organized according to the following sections: 

• This introductory section presents an overview of the ASEP program including 
descriptions of the nine ASEP projects, and an overview of the evaluation issues and 
questions; 

• Section 2 contains details on the formative evaluation methodology; 

• Section 3 presents findings according to the main formative evaluation issues and 
questions; and 

• Section 4 contains an overview of the main findings and conclusions. 

1 .1 Overview of ASEP Program 

The five-year, $85M ASEP initiative is a targeted Aboriginal s ki lls development program 
designed to promote maximum employment for Aboriginal people on major economic 
developments through a collaborative partnership approach, leading to lasting benefits for 
Aboriginal individuals, communities and fa mi lies. It is designed to address a broad spectrum 
of s ki lls and learning needs and provide access to jobs. The ASEP program is a part of the 
Government of Canada’s commitment in the 2002 Speech from the Throne to help Aboriginal 
people participate in economic development opportunities. 

The ASEP program has funded nine diverse projects in resource-based sectors, including 
mining, forestry, oil and gas, hydro development, fishery, and construction. Each project is 
located in a different community or group of communities with varying project scopes. 
Funding levels range from approximately $3M to $22M. The nine projects are located across 
the country in rural and urban areas, as well as northern and southern settings. 

The ASEP program was built on a recognition of the needs to be met in Aboriginal human 
resource development, and the practices and principles that have been identified in past policy 
developments and initiatives. 

The immediate objectives of the ASEP program are to: 

• Foster collaboration, partnerships, alliances and networks between government, the private 
sector, non-government agencies and Aboriginal groups to maximize the employment 
opportunities available to Aboriginal people in areas of major economic development by 
leveraging investment of resources, in-kind and financial, from ASEP project partners; 
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• Build the capacity of communities to address human resources development needs related 
to economic opportunities; and 

• Increase the quantity of employment-related skills training available to Aboriginal 
people in areas of major economic development beyond the level possible by reliance 
solely on Aboriginal Human Resources Development Agreement (AHRDA) funding. 

The long term objectives of the ASEP program are to: 

• Increase the number of Aboriginal people employed in the Canadian labour market; 

• Reduce the barriers to employment for Aboriginal people such as low educational 
attainment and lack of employment opportunities; and 

• Address a broad spectrum of skills and learning needs and provide access to jobs. 

According to the program logic model (see Appendix A), the main activities of the program 
include: developing and supporting partnerships for Aboriginal employment; and managing 
contribution agreements. The main outputs arising from these activities include: partnership 
agreements for the development of Human Resources (HR) plans; contribution agreements; 
and an increased capacity to address employment issues facing Aboriginal people. The 
anticipated outcomes are stated as: 

• Tailored skills enhancement/work experience opportunities ( immediate outcome ); 

• Enhanced employability of Aboriginal people ( intermediate outcome ); and 

• Long term sustainable employment for Aboriginal people ( ultimate outcome). 

In the most recent budget (March 2007), the Government of Canada announced that it will 
extend and expand the ASEP program by providing an additional $105 million over the next 
five years, including $35 mil lion in the first two years. 



1 .2 Summary of ASEP Projects 

The nine ASEP projects range in size, scope and activities. Each of the nine projects is 
described briefly below, 1 with summary information provided in Tables 1 and 2. 



1.2. 1 VanASEP Construction Careers Project 

The VanASEP project is a partnership that was formed with the aim of meeting the s kil ls 
shortage of the construction industry by creating opportunities for Aboriginal people to explore 
careers in construction. Aboriginal people will gain employability s kil ls while working on 
opportunities relating to the Vancouver Trade and Convention Centre project, the Vancouver 
Port Expansion project, the Sea to Sky Highway project and other major construction projects in 



1 Information for these summaries was derived from the project profiles presented on the ASEP website: 
http://www.hrsdc.gc.ca/en/employment/aboriginal training/proiects/indcx.shtml 
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the Lower Mainland. Partners in this initiative include the First Nations Employment Society, 
Aboriginal Community Career Employment Services Society, the Metis Provincial Council of 
British Columbia (BC), the Tsawwassen First Nation, the Spo7ez Society - representing the 
Squamish Nation and Lil'wat Nation, the BC Construction Association, the Vancouver Regional 
Construction Association, the Vancouver Port Authority, PCL Construction Ltd., Houle Electric 
Co., Lockerbie & Hole, Peter Keiwit & Sons, and the Province of British Columbia. 
The contribution agreement is held with the VanASEP Training Society. 



1.2.2 Aboriginal Mine Works 

Aboriginal Mine Works acts as a bridge between Aboriginal people in Alberta, the service 
providers and full-time employment in the oil sands industry. A five phase program is offered 
to prepare individuals for work. Elements of the program include pre-training, academic 
upgrading and occupational work experience. Partners in this initiative include the Athabasca 
Tribal Council, the Metis Nation of Alberta, Exxon Mobile, Syncmde, Suncor, Long Lake 
Project, ALP AC, Albian Sands, Petro-Canada, Canadian Natural Resources Limited, Japan 
Canada Oil Sands Limited, ATCO Group, Conoco, Deer Creek, Enbridge, Encana and the 
Government of Alberta (Human Resources and Employment). The contribution agreement is 
held with Wood Buffalo Partners in Aboriginal Training Association. 



1.2.3 Trade Winds to Success 

The Trade Winds to Success Project aims to improve employment opportunities for 
Aboriginal people by delivering pre-apprenticeship training that could lead to meaningful 
employment in Alberta's construction industry. The training-to-employment program offered 
by the Trade Winds to Success Training Society aims to provide the opportunity for 
Aboriginal people to gain employability skills by offering life s ki lls courses, academic 
upgrading to write the trades entrance exam level four, and eight weeks of union shop and 
hand s ki lls training. Partners in this initiative include the Oteenow Employment & Training 
Society, the Metis Nation of Alberta, the Treaty Seven Economic Development Corporation, 
the International Brotherhood of Boilermakers Local 146, the International Association of 
Bridge, Structural, Ornamental and Reinforcing Ironworkers, Machinery Mover, Riggers and 
Welders, the Alberta and Northwest Territories (District of MacKenzie) Regional Council of 
Carpenters and Allied Workers, the United Association of Journeymen and Apprentices of the 
Plumbing and Pipefitting Industry of the United States and Canada Local 488, Ironworkers 
Apprenticeship and Training Plan Local Unions 720 and 725, and the Edmonton Pipe Trade 
Education Trust Fund. The contribution agreement is held with Trade Winds to Success 
Training Society. 
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1.2.4 Northwest Territories Industrial Mining Skills 
Strategy 

The Northwest Territories Industrial Mining Skills Strategy aims to improve employment 
opportunities for Aboriginal people by providing education, job awareness, industrial s kil ls 
development and long-term employment opportunities in the diamond mining industry of the 
Northwest Territories. The Northwest Territories Industrial Mining Skills Strategy aims to 
provide Aboriginal people with access to pre-apprenticeship and apprenticeship programs, 
including underground mine training, that could lead to meaningful jobs in the industry. Partners 
in this initiative include the Yellowknife’s Dene First Nation, the Dogrib Treaty 11 Council 
(Tli Cho Government), the Lutsel K'e Dene Council, the North Slave Metis Alliance, Diavik 
Diamond Mines Inc., De Beers Canada Mining Inc., BHP Billiton, and the Government of the 
Northwest Territories (Department of Education, Culture and Employment). The contribution 
agreement is held with the Mine Training Society. 



1.2.5 Northwest Territories Oil and Gas ASEP 

The NWT Oil and Gas ASEP Skills Development Strategy aims to improve employment 
opportunities for Aboriginal people by providing s ki lls development and employment support 
leading to long-term employment opportunities in the oil and gas industry of the Northwest 
Territories. Programs include basic skills and pre-employment, industrial s ki lls development 
and employment support. The project is operating within the traditional territories of the 
Inuvialuit, Gwich'in, Sahtu and Deh Cho regions. Partners in this initiative include Deh Cho 
First Nations, Sahtu Dene Council Gwich'in Tribal Council, Inuvialuit Regional Corporation, 
Mackenzie Gas Project proponents and the Government of the Northwest Territories 
(Department of Education, Culture and Employment). The contribution agreement is held 
with the Aboriginal Futures Society. 



1.2.6 Manitoba Hydro Northern Training and 
Employment Initiative 

The Manitoba Hydro Northern Training Employment Initiative aims to improve employment 
opportunities for Aboriginal people in Northern Manitoba by providing s ki lls development, 
on-the-job work experience and long-term employment opportunities for the province's largest 
hydroelectric development project in two decades. Participants have the opportunity to be 
trained in designated and non-designated trades, as well as construction support occupations 
for self-employment in business and management. Partners in this initiative include the 
Nisichawayasihk Cree Nation, the Tataskweyak Cree Nation, the War Lake Cree Nation, 
the Fox Lake Cree Nation, the York Factory First Nation, the Manitoba Metis Federation Inc. 
(MMF), the Manitoba Keewatinook Ininew Okimowin (MKIO), Indian and Northern Affairs 
(INAC), Western Economic Diversification (WED), Manitoba Hydro, and the Province of 
Manitoba. In addition, the consortium has created partnerships with more than 20 training and 
educational institutions, as well as Manitoba's private sector. The contribution agreement is 
held with the Wuskwatim & Keeyask Training Consortium. 
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1.2. 7 Victor Diamond Mine Project 

The Victor Diamond Mine Project aims to improve employment opportunities for Aboriginal 
people by providing ski ll s development, on-the-job work experience and long-term employment 
opportunities with the Victor Diamond Mine Project in Northern Ontario. It offers various tools 
that include job information workshops, career counseling, training and work experience 
interventions, permanent job placements, and retention and advancement programs. Partners in 
this initiative include De Beers Canada and the Attawapiskat, Fort Albany, Kashechewan, 
Moose Cree, and Weenusk First Nations. The contribution agreement is held with James Bay 
Employment and Training. 



1.2.8 People, Land and Opportunities 

The People, Land and Opportunities project aims to improve employment opportunities for 
Aboriginal people by providing skills development, on-the-job work experience and long-term 
employment opportunities in the New Bmnswick forestry industry. The project offers a 
training-to-employment program that prepares Aboriginal people for careers in forestry and 
provides skills upgrading to individuals who already have experience related to the industry. 
Screening and job shadowing is also on hand to match candidates with meaningful 
employment opportunities. Partners in this initiative include the First Nations Human 
Resources Development Corp., Mawiw Tribal Council, New Bmnswick Aboriginal People 
Council, North Shore Micmac District Tribal Council, St. John River Valley Tribal Council, 
New Bmnswick Forest Products Association, Natural Resources Canada, and the Province of 
New Bmnswick (Aboriginal Affairs Secretariat). The contribution agreement is held with 
ASEP-NB Inc. 

1.2.9 Long Term Training Initiative for Nunavut’s 
Fishing Industry 

The Long Term Training Initiative for Nunavut’s Fishing Industry aims to improve 
employment opportunities for Aboriginal people by providing training programs, case 
management and retention activities to prepare individuals for long-term career opportunities 
in the Nunavut fishing industry. The project aims to assist the Inuit of Nunavut to obtain 
maximum employment and economic benefits from the development of a Nunavut-based 
fishing industry. Training includes basic upgrading and lessons on career opportunities in the 
fishing industry, and diploma program training. An at-sea mentorship program will also be 
developed and implemented to increase job retention. Partners in this initiative include the 
Nunavut Tunngavik Incorporated, Kakivak Association, Hunters and Trappers Association, 
Baffin Fisheries Coalition, Government of Nunavut (Department of Environment and 
Department of Education), and the Nunavut Arctic College. The contribution agreement is 
held with Nunavut Fisheries Training Consortium. 
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1.2.10 Summary of Planned Contributions and Results 
Targets 

As illustrated in Table 1 below, the HRSDC contribution to the nine projects ranged in size 
from $2.9M to $22.0M. Similarly, there was a large variation in the planned leveraged 
contributions for the projects ranging in size from $1.0M to S40.0M. According to the 
contribution agreements, the contributions from HRSDC will total $75. 9M across the nine 
projects with additional S90.5M as planned leveraged contributions. This result in total 
contributions to the nine projects predicted at $166.4M. 



Table 1 

Summary of Planned Contributions 


ASEP Project 


Province 

Territory 


HRSDC 

Contribution 


Planned 

Leveraged 

Contributions 


Total Planned 
Contributions 


VanASEP Construction 
Careers Project 


BC 


$7.8M 


$13. 8M 


$21. 6M 


Aboriginal Mine Works 


AB 


$3.3M 


$1.5M 


$4.8M 


Trade Winds to Success 
Project 


AB 


$3.9M 


$1.6M 


$5.5M 


NWT Industrial Mining 
Skills Strategy 


NWT 


$14. 9M 


$25.1 M 


$40. 0M 


NWT Oil and Gas ASEP 


NWT 


$1 0.0M 


$2.7M 


$12.7M 


Manitoba Hydro Northern 
Training and Employment 
Initiative 


MB 


$22. 0M 


$40. 0M 


$62. 0M 


Victor Diamond Mine 
Project 


ON 


$7.9M 


$2.7M 


$10.6M 


People, Land and 
Opportunities Project 


NB 


$2.9M 


$1.0M 


$3.9M 


Long Term Training 
Initiative for Nunavut’s 
Fishing Industry 


NU 


$3.2M 


$2.1 M 


$5.3M 


TOTAL 


$75.9M 2 


$90.5M 


$166.4M 


Source: Contribution Agreements for ASEP projects 



Within the contribution agreements signed for each project, there is an outline of targeted 
results for the entire funding period covered by the agreement. Table 2 contains a 
summary of the targeted results according to each project. Overall, under the agreements, 
it is planned that 9,976 individuals will be assessed for ASEP interventions, 6,245 will 
participate in interventions and 4,113 participants will complete their action plans. It is 
anticipated that 2,021 participants will become employed with the employers targeted 



2 



The difference between the $85M in Section 1.1 and the S75.9M indicated here are the administrative costs of delivering 
the ASEP program. 
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under the agreement, with a similar proportion (1,834) returning to employment 
elsewhere. Slightly over $23M is anticipated in short term income support savings. 



Based on information from Table 1 and Table 2, the planned per participant cost for the 
4,113 participants targeted to complete action plans would be $40,457 per participant 
overall, of which $18,454 would be HRSDC contribution. The planned cost per participant 
for those who will become employed with targeted employers or employment elsewhere 
would be $43,165 per participant overall, of which $19,689 would be HRSDC contribution. 



Table 2 

Summary of Targeted Results over Entire Funding Period 


ASEP Project 


Province 

Territory 


Target 
Number of 
Individuals to 
be Assessed 


Target 
Number of 
Individuals 
to 

Participate 


Target 
Number of 
Individuals 
to 

Complete 
an Action 
Plan 


Target 
Number of 
Individuals 
to be 
Employed 
with 

Targeted 

Employers 


Target 
Number of 
Individuals 
to Return to 
Employment 
Elsewhere 


Target 

Short 

term 

Income 

Support 

Savings 3 


VanASEP 
Construction 
Careers Project 


BC 


900 


600 

(67%) 


600 

(67%) 


300 

(33%) 


250 

(28%) 


$500K 


Aboriginal Mine 
Works 


AB 


300 


120 

(40%) 


120 

(40%) 


104 

(35%) 


120 

(40%) 


No target 


Trade Winds to 
Success Project 


AB 


380 


224 

(59%) 


60 

(1 6%) 


54 

(14%) 


54 

(14%) 


$3K 


NWT Industrial 
Mining Skills 
Strategy 


NWT 


1,500 


1,100 

(73%) 


380 

(25%) 


380 

(25%) 


380 

(25%) 


No target 


NWT Oil and 
Gas ASEP 


NWT 


1,366 


1,366 

(100%) 


1,093 

(80%) 


51 

(4%) 


547 

(40%) 


No target 


MB Hydro 
Northern 
Training and 
Employment 
Initiative 


MB 


2,631 


1,115 

(42%) 


807 

(31%) 


794 

(30%) 


100 

(4%) 


$19,395K 


Victor Diamond 
Mine Project 


ON 


1,530 


1,020 

(67%) 


683 

(45%) 


190 

(12%) 


153 

(10%) 


$2,074K 


People, Land 
and 

Opportunities 

Project 


NB 


800 


500 

(63%) 


220 

(28%) 


92 

(12%) 


220 

(28%) 


$1 ,120K 


Long Term 
Training 
Initiative for 
Nunavut's 
Fishing Industry 


NU 


569 


200 

(35%) 


150 

(28%) 


56 

(10%) 


10 

(2%) 


No target 


TOTAL 


9,976 


6,245 

(63%) 


4,113 

(41%) 


2,021 

(20%) 


1,834 

(18%) 


$23,092K 


Source: Contribution Agreements for ASEP projects 



3 Short term income support savings refers to the difference between the aggregate of each participant's entitlement to 
social assistance benefits and the actual pay out of the benefits to that participant. 
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1.3 Profile of Participants 

A profile of participants 4 was developed from case management information supplied by the 
nine projects. The main participant characteristics identified based on available participant 
data included: 

• Gender: Approximately three quarters of participants (76%) were male. 

• Age: Approximately two-thirds were under the age of 35. 

— Under 25 years old - 32% 

— 25 to 34 years old - 32% 

— 35 to 44 years old - 25% 

— 45 years or older - 1 1% 

• Education: The majority of participants had not graduated from high school. 

— Grade 8 or less - 12% 

— Grade 9 to 1 2 (no diploma) - 5 1 % 

— High school graduate - 32% 

— Some Vocational or post-secondary - 4% 

• Aboriginal group : Most participants were members of First Nations. 

— First Nations - 80% 

— Metis -10% 

— Inuit - 10% 

1.4 Evaluation Objectives and Issues 

The Results-based Management and Accountability Framework (RMAF) developed for the 
ASEP program 5 guides the continuous monitoring and evaluation process. The continuous 
monitoring and evaluation of the ASEP program is a condition set out in Section 1 (Schedule 
E: General Conditions) of each of the contribution agreements between the Minister of 
HRSDC and the respective ASEP project partners, which specifically states: 

Canada will conduct an evaluation during the period of the Agreement to 
determine the success of the Proposal activities that are being supported 
under the Agreement. The Proponent shall cooperate with Canada in the 
conduct of the evaluation by collecting and providing such information as may 
be reasonably requested by Canada in relation to the Strategy activities. 



4 This profile is based on incomplete data (approximately 2,000 cases) because 20% of participant's information was 
missing. As a result, the profile should be used for providing a general snapshot of participants rather than providing 
specific details. 

5 Results-based Management and Accountability Framework (RMAF) for the Aboriginal Skills and Employment 
Partnership (ASEP) program - Human Resources and Skills Development Canada, 2003. 
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The overall objectives of the formative evaluation were to assess the rationale, relevance, 
implementation, results to-date, and accountability of the ASEP program. At the formative 
stage, five case studies were implemented to contribute to the understanding of the context in 
which the ASEP program is operating, the achievements made related to partnership building, 
and the early impacts being made in community capacity building related to human resource 
development. 

The ASEP RMAF outlined the evaluation issues and questions to be addressed by the 
formative evaluation. It is noted that while many of the questions in the framework are 
summative in nature and would normally be addressed towards the end of an initiative, 
the intention of the formative is to report on progress towards, and l ik elihood of achievement 
of, the outcomes indicated in the framework. The issues and questions addressed by the 
formative evaluation are included in Table 3 below. 



Table 3 

Formative Evaluation Issues and Questions 6 



Issue: Program rationale and relevance 

Q1 - Is there a continued need for the program? 

Q2 - To what extent does the ASEP program reflect broad federal government Aboriginal policy 
directions? 

Issue: Program implementation 

Q3 - Have the ASEP projects been implemented as planned? 

Q4 - Was the accountability structure established as set out in the RMAF? 

Q5 - Were the services appropriate to the target clients? 

Q6 - Were funds expended as set out in planning documents? 

Q7 - How was capacity of the governing structures/delivery agents developed? 

Q8 - Were targeted clients and communities reached? 

Q9 - How well did the governing structures/delivery agents coordinate with other employment 
related services for Aboriginal workers? 

Issue: Program results to-date 

Q1 0 - To what extent have ASEP contributions increased the ability of organizations to address the 
needs of Aboriginal populations? 

Q1 1 - To what extent have ASEP funded projects and new knowledge generated resulted in 
improved government policies and services for Aboriginal populations? 

Q13 - To what extent has the ASEP program resulted in skills enhancement and work experience 
opportunities for Aboriginal people that resulted in increased ability to participate in the 
community, compete in the labour market and obtain and maintain employment? 

Issue: Accountability 

Q16- Are adequate safeguards in place to ensure that federal investments do not duplicate or 
displace non-federal investments? 

Q17 - Has the ASEP program leveraged additional investments from project partners? 

Q18 - Are ASEP investments incremental to existing non-federal investments? 



6 The issue of cost-effectiveness will be addressed in the summative evaluation. The cost-effectiveness issue includes 
both Q14 - Are ASEP projects cost ejfective?, and Q15 - How do ASEP funded projects compare to other similar 
programs in this field?. As well, an additional question under the Program results to-date issue will be added for the 
summative evaluation (Q12 - Has ASEP contributed to increased capacity to address employment issues facing 
Aboriginal people?). 



Formative Evaluation of the Aboriginal Skills and Employment Partnership Program 






Formative Evaluation of the Aboriginal Skills and Employment Partnership Program 




2. Evaluation Methodology 



The methodology developed and implemented for the formative evaluation included: 

• Case studies with five Aboriginal Skills and Employment Partnerships (ASEP) projects 
which included key informant interviews, discussion groups, a project document review, 
and socio-economic profiles, and 

• Assessment and analysis of client case management and administrative data for all of 
the nine ASEP projects. 

An overview of these methods is included below. The details for each method are provided in 

the various technical reports prepared for the evaluation. 



2.1 Case Studies 

Case studies were conducted for the following five projects: 

• VanASEP Construction Careers Project; 

• Aboriginal Mine Works; 

• NWT Oil and Gas ASEP; 

• Manitoba Hydro Northern Training and Employment Initiative; and 

• People, Land and Opportunities Project. 

These five case-studies were chosen in partnership with ASEP Program Management and 
with consideration of the following three factors: size of the contribution agreement, 
geographical distribution, and rural/urban location. Each case study consisted of a document 
review, key informant interview, discussion groups, and socio-economic profiles. A separate 
technical report was prepared for each of the five projects. 



2. 1. 1 Document Review 

The objectives of the document review were to assist in understanding the context of the 
various projects and to address some evaluation issues. Documents reviewed included policy 
documents, annual reports, plans, administrative forms and promotional materials for 
each project. 
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2. 1.2 Key Informant Interviews 

The objective of the key informant interviews was to gather in-depth information including 
views, explanations, examples and factual information that addressed specific evaluation 
issues. The key informant interviews should be viewed as a complement to the other lines of 
evidence gathered as part of this evaluation, in that they provide supporting qualitative 
information. The evaluation team worked in conjunction with the various project contacts to 
identify appropriate key informants for the evaluation, including project partners, project 
staff, industry representatives and employers, staff from training institutions and Aboriginal 
organizations. 

The evaluation team was provided with the lists of potential key informants and the selection 
ensured representation of ASEP partners, employers, staff, training institutions and Aboriginal 
organizations involved with each project. It should be noted that the lists were not necessarily 
exhaustive. Approximately 10 interviews were conducted in each of the five case study 
communities. In total, 67 interviews were conducted with key informants. The distribution 
according to sub-groups was: 

• Project partners (n=35) 

• ASEP project staff (n=14) 

• Industry representatives/employers (n=8) 

• Training institutions and Aboriginal organizations (n=10) 

Respondents were provided with a copy of an introductory letter from Human Resources and 
S kil ls Development Canada (HRSDC) outlining the purpose of the evaluation, how the collected 
information would be used, and contact information if they had any questions. Interviews were 
conducted using semi- structured guides with open-ended questions that were tailored according 
to the sub-groups. Where relevant, the versions of the guides shared similar questions to address 
the evaluation issues. All interview guides were pre-tested within the first case study undertaken. 
The notes from interviews were grouped according to evaluation questions and issues using an 
evidence matrix. 



2.1.3 Discussion Groups 

The objective of the discussion groups was to gather qualitative information from participants 
and community members with respect to specific evaluation issues. The discussion group 
method was designed to complement the other qualitative and quantitative methods used for 
the formative evaluation. The discussion groups were conducted as a facilitated group 
discussion that leads to an exploration of the issues and understanding of the context behind 
individual’s experiences. 

Approximately two discussion groups were conducted in each of the five case study 
communities. Discussion groups were arranged in consultation with the ASEP project staff 
and the participating communities. Given the nature of the projects, ASEP project staff or 
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training institutions recruited individuals to participate in the discussion groups in most 
communities. This may have introduced a bias to the discussion group process given that the 
participants were not randomly selected and recruited by the evaluation team in most instances. 

The discussion group guides were developed to address specific evaluation questions. 
The guide was pre-tested in the initial case study. Participants were offered a $50 honorarium 
for their participation. Notes were taken in each group by a member of the evaluation team 
while the other team member facilitated the group. The notes were sorted according to 
question. The analysis ascribed importance to responses based on consistency of responses 
across discussion group participants. 

In total, 12 discussion groups were conducted with 92 participants, including 67 ASEP clients 
and 25 community members, trainers, and employers. 



2. 1.4 Socio-economic Profiles 

Socio-economic profiles were developed for the communities involved in the five ASEP 
projects selected for case studies. The participating communities and regions covered by the 
projects were identified from the contribution agreements for each of the projects. Profiles 
were developed using publicly available data including Census data (1996; 2001), 
the Aboriginal Peoples Survey (2001), and information from Indian and Northern Affairs 
Canada (ENAC) such as the community well-being indices. 



2.2 Assessments of Client Data and Data Systems 

The data assessment component was a stocktaking of data systems as well as the processes of 
using these systems for client service and program tracking. It provided an early and 
comprehensive assessment of how well the data can be expected to match the requirements for 
monitoring, as well as the formative and summative evaluations for each of the nine ASEP 
projects. This process assisted in addressing issues of implementation and accountability. This 
assessment also provided guidance to HRSDC and ASEP projects to address any gaps and 
limitations. 

The data assessment component was designed to ensure that all parties understand to what 
extent the client data and data systems can realistically contribute to the evaluation 
information needs, both in the current state and with any remedial work that is indicated from 
the assessment. 

The ASEP data assessments were designed to: 

• Assess the capacity of ASEP project administrative data files to support evaluation; 

• Determine the extent to which data not available from administrative data files are 
available from other sources; 

• Map the series of steps required to obtain the data needed for the evaluation; 
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• Assess the adequacy of data systems to meet monitoring and evaluation needs; 

• Provide a summary and analysis of available client program data to date to assess the 
experiences of program clients; and 

• Propose solutions for addressing any data gaps or constraints. 

Nine data assessment reports were completed and shared with ASEP program management 

and ASEP projects for remedial work. 

The key components of the data assessment included: 

• Project Initiation and Site Visit: For each project, the initial task was to contact the Project 
Coordinator, review the letter sent by HRSDC informing them of the evaluation, discuss the 
need to access their files, and determine the most appropriate location and time for a site 
visit. Member(s) of the Aboriginal Employment Services (AES) Inc. research team then 
visited at least one site for each of the nine ASEP projects between September 25, 2006 and 
March 23, 2007. With the assistance of project staff, business processes were documented 
and case management and financial data was reviewed on-site. In addition, a copy of case 
management data was provided for later analysis. 

• Data Collection Process Description : An important element of the data assessment was 
documenting the data collection process to help understand how, when, and in what format 
data is captured. This helped the research team decide how best to select cases for review, 
identify any process weaknesses that may result in data gaps, and determine where data is 
held and in what format. Data collection processes were documented through interviews 
with project staff. 

• Data Mapping: The data mapping process matches the evaluation issues and indicators with 
the data sources available. For the ASEP data assessments, the data mapping concentrated 
on questions to be answered using case management and administrative data. Data mapping 
was done by identifying evaluation questions that rely to some degree on case management 
and/or other administrative date, identifying data required to develop the client survey 
frame, developing a list of data requirements related to case management and administrative 
data, identifying the source of the data, identifying the location of the data (where it is 
physically held), and identifying the data elements that are important from a monitoring and 
evaluation perspective. Once this process was complete, gaps between data available from 
administrative sources and data needed for evaluation were identified. Other sources of data 
were then sought to fill the gaps. 

• Data Assessments: After identifying the data requirements and the sources of data available 
to meet these requirements, it was necessary to examine the integrity, quality, reliability, 
validity and logic of all data variables for all nine ASEP projects. Specifically, the data 
assessments clearly identified the data available to support the evaluation; location and 
integrity, quality, reliability, validity and logic of data variables according to life-cycle of 
program or intervention; age, frequency and timeframes covered by data files; gaps and 
limitations that currently exist (and the reasons underlying the gaps) in the data; 
and potential risks to the evaluation study as they pertain to gaps in data availability, validity 
and quality. 
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Gap Analysis: Data gaps identified through the data mapping process or as a result of data 
assessments were brought together and recommendations for how best to address these gaps 
were developed for each ASEP project. 

Profile of Participants : To provide additional context for the data assessment, using 
available administrative data, the research team has profiled ASEP program participation as 
of the date of the site visit for each of the nine projects. This profile included number of 
clients by age, sex, education level, and Aboriginal group; program participation by age, 
sex, education level, and Aboriginal group for each type of intervention; and action plan 
results by age, sex and education level. 
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3. Findings 



This section presents the findings according to the evaluation issues and questions. After each 
set of findings, supporting evidence is provided from all lines of inquiry. 



3.1 Program Rationale and Relevance 



Q1 - Is there a continued need for the program? 

There is a strong continued need for the Aboriginal Skills and Employment Partnership 
(ASEP) program, as demonstrated through the case studies and the socio-economic 
profiles. The ASEP program is viewed as addressing needs of both the Aboriginal 
communities and industry. Specifically, ASEP interx’entions are perceived as relevant to 
the needs identified within Aboriginal communities to develop skills and gain work 
experience in order to fully participate in key industrial projects in their regions. 



An overview of the socio-economic profiles developed from 2001 Census data and Aboriginal 
Peoples Survey data demonstrates that the Aboriginal communities participating in the ASEP 
program tend to have greater socio-economic disadvantages when attempting to participate in 
key industrial projects in their regions when compared with non-Aboriginal populations. 
Inmost cases reviewed, Aboriginal people had significantly lower levels of education, 
employment and earned income when compared to non-Aboriginal people. Given the labour 
requirements of these large industrial projects associated with the ASEP program, there is a 
need to assist Aboriginal people to upgrade and develop their skills and work experience to 
meet these requirements in order to fully participate in the work force related to the industrial 
projects. With involvement from industry, ASEP projects are designed to ensure that 
Aboriginal participants receive training that is relevant to industry requirements. 

Aboriginal and industry partners consistently indicated that there is a need for the ASEP 
projects. Emphasis on a demand driven model was viewed as particularly relevant for the 
contexts of the specific ASEP projects. The ASEP program was also viewed by Aboriginal 
and industry partners as meeting the need for supporting partnerships in order to accomplish 
project objectives. The need for effective, quality partnerships between various Aboriginal 
communities and industry was reiterated during interviews with partners. In some instances, 
the Aboriginal partners brought together for the ASEP project indicated that this was the first 
time that the specific groups making up the partnership had actually come together to jointly 
work on issues affecting their communities. The ASEP program’s emphasis on partnerships 
was viewed as assisting in addressing this need. 

Partners, participants, community representatives and training organizations viewed the ASEP 
projects as addressing specific needs of participants and communities. The main needs identified 
during interviews and discussion groups as being addressed by the ASEP program included: 

• Development of s kill s, including occupation-specific s kill s, life skills and essential skills; 
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• Raising awareness of trades among participants and the Aboriginal communities more 
generally; 

• Obtaining work experiences and “hands-on” training; 

• Raising awareness of workplace requirements, work ethic, and work habits; 

• Obtaining certification and apprenticeship hours; 

• Providing community-based training, when available; 

• General education upgrading; 

• Preparing for jobs that are not just unskilled labour jobs; and 

• Providing financial support while training and gaining work experience. 

The documents reviewed for each case study project demonstrated the various needs that the 
specific ASEP project was designed to address. The project proposals clearly outlined in each 
case the purpose of the projects, and the rational as to why this project was needed at the time 
of application. With one exception where there was a sudden, sharp down turn in an industry 
and the need for s ki lled workers in the sector was greatly reduced, all other contexts examined 
indicated that the same needs identified in the original project proposals remained from both 
the perspectives of Aboriginal communities and industry. 



Q2 - To what extent does the ASEP program reflect broad federal 
government Aboriginal policy directions? 

The ASEP program continues to reflect the broad fedeml government Aboriginal 
policy directions as outlined in recent planning and policy documents. 



The most recent Report on Plans and Priorities (2006-2007) for Human Resources and Social 
Development Canada (HRSDC) has identified building more effective partnerships to improve 
Aboriginal labour market outcomes as being directly linked to the achievement of the strategic 
outcome of “enhanced Canadian productivity and participation through efficient and inclusive 
labour markets, competitive workplaces and access to learning”. Within the plan there is the 
recognition that the achievement of strategic outcomes will require that HRSDC “works closely 
with provincial and territorial partners, Aboriginal people, not-for-profit and community-based 
volunteer organization, and with Canadians, building on established relationships bilaterally 
and through multilateral forums”. The ASEP program, with its emphasis on partnerships and 
results based on the direct labour market participation of Aboriginal people in key industries, is 
directly supporting the federal government priorities. 

Other policies and priorities that provide rationale and program authority for the ASEP 
program include: 

• Aboriginal Skills and Employment and Inclusion: In the 2006 Speech from the 
Throne the government committed to “seek to improve opportunity for all Canadians, 
including Aboriginal people and new immigrants”. 

• Skills and Learning: A commitment to skills and learning is at the foundation of 
Canada’s Innovation Strategy, which was launched in February, 2002. Under this 
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Strategy, the Government of Canada indicated it would consider, in cooperation with 
the provinces, territories and other partners, a targeted skills development initiative for 
Aboriginal people. The Reference Group of Ministers on Aboriginal People (RGMAP) 
identified a wide range of possible initiatives to respond to the Government of 
Canada’s commitment to Aboriginal people, which included the ASEP program. 

• Social Union Framework Agreement (SUFA): Under the SUFA signed in 1999, 
the First Ministers agreed to promote the full and active participation of all Canadians 
in Canada’s social and economic life. The SUFA also states that “governments will 
work with the Aboriginal peoples of Canada to find practical solutions to address their 
pressing needs.” 

In the most recent budget (March 2007), the Government of Canada announced that it will 
extend and expand the ASEP program by providing an additional $105 million over the next 
five years, including $35 million in the first two years. 



3.2 Program Implementation 



Q3 - Have the ASEP projects been implemented as planned? 

Overall, most activities under the various ASEP projects have been implemented as 
origincdlv planned. Information collected from project documents, partners and staff 
demonstrated that key assessment and training activities had been undertaken and 
most planned components were undem’ay at the time of the interviews. 



The review of documents collected during the case studies indicated that in most cases, 
implementation of activities has proceeded according to plan. A comparison of annual reports 
with contribution agreements resulted in very few differences. Partners and project staff 
indicated that where there were differences, this was most often due to one or more of 
the following: 

• Differences in timing due to unanticipated delays in some activities starting up 
(e.g., finding an appropriate training institution, adapting materials for the Aboriginal 
context, working out the assessment procedures with multiple parties, etc.); 

• Differences in timing due to delays in timeline for the associated industrial project or 
sudden changes in the targeted sector; 

• Differences in timing due to the different paces and starting times of different 
Aboriginal partners within a project; 

• Differences in scopes of activities. Scopes of activities were adjusted to usually include 
additional components given additional requirements for participants (e.g., upgrading 
of basic/essential skills, housing support) or as a result of suggestions from industry 
(e.g., specific certifications, expanding scope to related sub-sectors). 
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Q4 - Was the accountability structure established as set out in the RMAF? 

Overall, there are significant gaps in the case management and financial data required for 
evaluation and accountability. In cdl but one of the nine projects, remedial work is needed 
to ensure that data accurately reflects project activity. 

The most serious gaps with respect to case management data are: 

• There is no electronic case management data available for a significant proportion 
of clients. As a result, complete case management data are neither readily available 
on-site, nor available from HRSDC’s Standard Data File (SDF). 

• Data stored in case management systems are miscoded in the majority of cases. 

The financial data, for most projects, are more complete; however, for cdl projects, work is 
required to link project costs to individual clients. 



The data assessment found that there is considerable work required for most projects to bring 
their case management and financial data up to a level where they would be assessed as 
adequate for evaluation and accountability purposes. The data mapping exercise demonstrated 
that, for the most part, the various case management and financial systems used by the nine 
projects have the required data fields to meet monitoring and evaluation requirements. The 
main challenges in the data were identified during the data assessment exercise when the 
actual case management data contained in the systems were assessed for coverage, validated 
with hard copy files, assessed according to interventions, and matched with the data available 
in the financial systems. 

The gap analyses conducted for the nine projects identified common gaps among the projects 7 
case management systems: 

• Clients who received an employment benefit or support measure but are not in electronic 
files uploaded to HRSDC’s SDF - Because these individuals are not in Contact IV or ARO 
Suite , 8 their records are not uploaded to HRSDC’s Standard Data File and consequently, they 
are not included in the internal accountability reporting process. The exclusion of these 
clients from Contact IV or ARO Suite means that they are excluded from any analysis of 
clients, resulting in an understatement of number of people benefiting from the ASEP 
program, and resulting in inaccurate profiles. The lack of information about individuals in 
Contact IV or ARO Suite may create bias in some evaluation methodologies. In particular, 
the analysis of administrative and participant survey data would be biased towards those 
participants whose records are actually within HRSDC’s Standard Data File. These may or 
may not be representative of the overall population of ASEP participants. The direction of 
this bias would not be easily determined without data on the overall population. 

• Clients who received an employment benefit but do not have a hard copy case 
management file - There are instances where there have been payments made according to 
the financial systems, but there is no hard copy case management file available for these 



7 

It should be noted that gaps were included when similar gaps were identified in two or more projects. Specific details of 
the gaps by project are contained in separate technical reports. 

8 Contact IV and ARO Suite are case management systems. 



Formative Evaluation of the Aboriginal Skills and Employment Partnership Program 





individuals. It is important for accountability and evaluation purposes that there exists a hard 
copy management file for each individual. 

• Missing and/or inaccurate case management data on client characteristics - When records 
in Contact IV were compared with hard copy files, there were data missing in Contact IV 
that were available in the hard copy files. Missing data on client characteristics affects the 
accuracy of client profiles and any evaluation sampling strategies. This gap could be 
addressed by additional data entry by the project staff. 

• Missing and/or inaccurate information on client interventions - When records in Contact 
IV were compared with hard copy files, many of the interventions were miscoded. 
For example, s kill s development interventions were often identified in Contact IV files as 
Job Creation Partnerships (JCP) or Aboriginal - Other. This gap results in inaccurate and 
misleading data being uploaded to HRSDC’s SDF, which in turn is then being used for 
accountability and evaluation purposes. 

• Failure to close action plans - In some cases, there were large proportions of action plans in 
Contact IV or ARO Suite that were left open. While some action plans could be legitimately 
left open as clients are currently participating in interventions, there were a number found 
where there had been no activity for over a year. It is important to close action plans once 
interventions are complete in order to capture action plan results for accountability and 
evaluation purposes. 

The most common gap identified among the projects financial systems included: 

• Failure to allocate costs back to clients - Costs for training, student allocations, etc. 
should be allocated back on a per client basis so that costs per intervention, as well as 
costs per client can be calculated. 



Q5 - Were the services appropriate to the target clients? 

Overall, the services were perceived as appropriate for the target clients. In many instances, 
training and work placements had been tailored to meet the specific needs of clients, and 
interventions had been adjusted based on evaluations and feedback received from 
participants. Even with the tailoring and adjustments, there was the acknowledgement that 
the ASEP projects could not accommodate some issues such as providing services for those 
who had very low education levels who requested training, and some contextual issues such 
as participants having to leave the community to find work until the large projects 
are underway. 



Overall, partners, ASEP project staff and representatives from training institutions indicated 
that the services were appropriate for the targeted clients. Key factors that were identified by 
these groups as contributing to assuring that the services were appropriate included quality of 
instructors, role of job coaches, input from industry on the types of training to be offered, 
variety of services offered, and flexibility within and across interventions. In many instances, 
respondents reported that there was ongoing feedback requested from students and instructors 
to assist in making appropriate adjustments to services offered to participants. Respondents 
also highlighted a number of challenges involved in assuring that the services were 
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appropriate, many of these outside of the specific scopes of the projects. Some of the most 
frequently cited challenges included: 

• Occurrence of “social passing ” in high schools - This involves passing students from one 
year to the next due to their age rather than their academic achievements. As a result, some 
respondents reported that potential clients are arriving with high school credentials but are 
being assessed at levels far below high school equivalences. There is a large gap then, that 
exists between their academic level and the level required for them to actively participate in 
ASEP designed interventions. 

• Challenges in finding adequate work experience opportunities - Respondents report that 
there are challenges in moving participants into appropriate work experience activities 
once they have received training, given that in some cases the actual industrial projects 
associated with the ASEP project are behind schedule, are not yet developed enough to 
start hiring ASEP participants, or there has been a sudden down-turn in the specific sector. 
As a result of these gaps in timing and opportunities, there are reported instances where 
ASEP participants have had to move out of their communities and provinces/territories to 
find adequate employment opportunities to collect apprenticeship hours, and/or gain work 
experience related to their training. It was anticipated among some of the participants that 
these apprenticeship and work experience opportunities would be available on the actual 
designated projects/sectors identified in the ASEP project (e.g., forestry, hydro project 
construction). 

Discussion groups with ASEP clients and community representatives provided similar 
assessment to key informants on the appropriateness of services. Overall, participants 
indicated that the services were appropriate and cited areas such as the quality of instructors, 
financial support available, and the variety of s ki lls developed. A number of areas were also 
identified as challenges. Some of the most frequently cited were: 

• More work placement opportunities to practice and apply techniques learned in 
classroom training situations; 

• Additional assistance with personal costs (e.g., transportation, child care); 

• Requirement to leave communities to get training and work experience; and 

• Redundant and/or inadequate assessment process. 



Q6 - Were funds expended as set out in planning documents? 

The actual expenditures, as reported in annual reports and summary financial reporting, 
were less than planned expenditures for the initial stages of the projects examined. This is 
to be expected given the delays encountered in implementing activities during the first 6 to 
12 months with many projects, and the nature of the activities during the start-up period. 
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The evaluation team received summary financial i nf ormation in the form of annual reports and 
summary financial reports for three of the five case study projects 9 . Based only on these three 
cases, the trend was that during the first stage of the projects, actual expenditures were 
significantly lower than planned expenditures. For the three projects examined, actual 
expenditures ranged from 34% to 60% of planned expenditures during the first 6 to 12 months 
of the project. Given the delays that some projects experienced during the start-up phase and 
the nature of the activities (e.g., assessment and training vs. wage subsidy) in the initial stages 
of the projects, these findings would be in line with what would be expected for similar projects 
in a start-up phase. 



Q7 - How was capacity of the governing structures/delivery 
agents developed? 

There was considerable variability in the amount of effort and resources the various ASEP 
projects had invested to date in developing the capacity of governing structures, ASEP 
staff and delivery agents. While two of the case study projects reported few activities in 
these areas beyond developing the basic by-laws and procedures associated with 
becoming an incorporated organization, two of the case study projects indicated that 
considerable effort had been made in developing capacity of delivery agents, ASEP staff 
and Boards of Directors. 



Responses from partners, ASEP staff and representatives from organizations delivering 
assessments and training varied considerably from project to project when describing the 
various activities undertaken to develop the capacity of governing structures and delivery 
agents. While some projects had invested considerable time, effort and resources in 
developing capacity of Board members, ASEP staff and delivery agents, others indicated that 
they had not yet had time to implement these activities, or did not see the rationale for 
implementing these activities given that capacity, particularly among ASEP staff, was one of 
the main hiring criteria. 

All projects participating in the case studies had developed by-laws, various policies and 
procedures, and job descriptions for the incorporated organization. In addition, some 
projects included capacity developing activities such as on-going training for the ASEP 
and delivery agent staff in various areas including assessment techniques, data systems, and 
case management. Some projects also presented information on how they had worked at 
developing the Board’s capacity by implementing Board retreats and planning sessions. 



9 ..... . 

The information obtained from project financial systems for the data assessment was not analyzed to determine the 
extent to which actual expenditures matched planned expenditures. As planned, data was used to assess the extent to 
which the financial systems were adequate with respect to monitoring and evaluation. 
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Q8 - Were targeted clients and communities reached? 

The ASEP projects have developed and implemented marketing, promotional and 
communication activities to ensure that targeted communities are reached. Activities 
included direct outreach via partners’ networks, community meetings, and direct 
promotional techniques such as flyers, newsletters, pamphlets and posters. The case 
studies identified challenges to ensuring that targeted communities are reached, with the 
main challenge identified as the relatively rapid turnover of staff within the various 
communities, which requires ongoing promotional and training activities to keep 
communities adequately informed. 



Partners, ASEP staff and community representatives reported various activities which the 
ASEP projects have undertaken to ensure that the targeted communities were aware of 
the ASEP project activities. Respondents reported that the partnership aspect of the ASEP 
projects contributed to increased awareness of ASEP activities among targeted communities, 
as communities tended to have representation on the Board of Directors for the ASEP 
organizations. Board members were often cited as having responsibility for informing 
their own communities of ASEP activities. Another key group identified as important in 
understanding the ASEP projects objectives and activities were the organizations 
providing assessment and referral activities to the ASEP program from the various 
communities (often Aboriginal Human Resources Development Agreement (AHRDA) 
holders). The challenge cited in keeping these organizations informed was the turnover 
rate among staff at these organizations, which required an ongoing promotional and 
training requirement for ASEP staff. 

A review of annual reports, marketing plans and promotional material provided by 
communities involved in the case study projects demonstrated that effort was being made 
by ASEP projects to inform targeted communities of their activities. Some communities 
were also able to provide data supporting the outcomes of their activities in this area. 
Three communities provided data in various documents demonstrating that they had 
attracted clients in key activities from each of their target communities. 

Given the issues with various projects’ administrative data outlined above (under the findings 
for Question 4), the client profiles developed from the administrative data provided to the 
evaluation team have major gaps, and are not likely representative of the clients who have 
obtained services from the ASEP projects. As a result, it is difficult to use these profiles to 
determine whether targeted clients are being reached. 
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Q9 - How well did the governing structures/delivery agents coordinate 
with other employment related services for Aboriginal workers? 

The quality of the coordination between the ASEP structures, delivery agents and other 
employment services for Aboriginal people seemed to vary considerably between projects. 
Key factors contributing to a successful coordination included a solid partnership with a 
shared vision, ongoing and good communication between partners and ASEP staff, solid 
case management systems, development and adoption of common tools and templates, and 
ongoing identification of common issues and areas of concerns. Where coordination has 
been perceived as less successful, issues identified included large staff turnover, 
overlapping services, and perceived competition between organizations. 



Partners, ASEP staff and representatives from Aboriginal organizations reported their 
perceptions of the extent to which the governing structures and delivery agents coordinated 
well with other employment related services for Aboriginal clients. The perceptions were 
relatively similar within projects, however varied considerably across projects. For some 
projects, respondents viewed the quality of coordination as very satisfactory. These people 
attributed the coordination to various factors, including: 

• A solid partnership in which partners share a common focus and vision. This common 
focus and vision is then relayed to the communities and other organizations via the 
partners so that everyone is clear on which services are provided under the ASEP 
program and why; 

• Ongoing, good communication between partners and ASEP staff. ASEP staff support 
community partners in their communication with their communities. This way, the 
organizations are not providing mixed or contradictory messages about ASEP activities 
and purpose; 

• Common case management systems that allow all relevant organizations to understand and 
be aware of which clients are receiving which services; 

• Development of common tools and templates so that organizations involved with ASEP 
clients at various stages (e.g., assessment and referral, training, work placement, etc.) are 
collecting the right information at each stage; and 

• Ongoing, good communication between ASEP staff and other organizations’ staff (i.e. 
AHRDAs) to ensure that issues and challenges are identified and addressed in a 
manner that is suitable to the various parties involved. 

In contrast, respondents from other ASEP projects indicated that there were challenges in the 
quality of coordination between ASEP projects and other organizations providing employment 
services to Aboriginal clients (i.e. AHRDAs). The challenges identified by the respondents 
included: 

• Competition among and between the organizations involved; 

• Overlap in services provided, particularly in the areas of assessment and screening; and 

• Large staff turnovers among both ASEP staff and other organizations’ staff. 
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3.3 Program Results To-Date 

Program success will be fully addressed in the summative evaluation. At the formative 
evaluation stage, multiple lines of evidence were used to provide some preliminary 
information on progress towards success, to have indications of the likelihood of the ASEP 
projects’ success in various areas. The reader should keep in mind that these findings are 
preliminary in nature, and can only be confirmed during the summative evaluation. 



QIO-To what extent have ASEP contributions increased the ability of 
organizations to address the needs of Aboriginal populations? 

In most cases, the ASEP projects were identified as increasing the ability of organizations 
to address the needs of Aboriginal populations. In a few cases, the projects were perceived 
to have limited success in this area by providing services that overlap with existing 
opportunities provided by cdready established programming. 



Partners, community representatives, and Aboriginal organization representatives were asked 
to provide information on the extent to which the ASEP program has contributed to an 
increased ability of organizations to address the needs of Aboriginal people. Most responded 
positively to this question by providing examples where they felt the ASEP program had been 
successful in this area. Common responses included: 

• Better understanding among organizations (including employers) of the challenges that 
young Aboriginal people face (e.g., literacy, alcohol/drug use, etc.); 

• Development of networks among service providers; 

• Addressing cultural barriers and challenges with employers; 

• Capacity to deliver training directly in the community; and 

• Providing a wider variety of services, supports, training, and employment opportunities. 

A smaller proportion of respondents reported that they perceived the ASEP projects as 
essentially duplicating existing services and opportunities provided under other programs. 



Q11 - To what extent have ASEP funded projects and new knowledge 
generated resulted in improved government policies and services 
for Aboriginal populations? 

There were a few examples provided during the evaluation that demonstrated how new 
knowledge generated from ASEP projects was beginning to be transferred and potentially 
incorporated into other related services. 
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The document review and key informant interviews identified a few examples where various 
aspects of the ASEP projects were being identified as successful and considered for inclusion 
in other models. In one case, the training model had been presented at an international 
conference and was being considered for implementation at the provincial level for non- 
Aboriginal clients. In another case, the partnership model developed for the ASEP project was 
the topic of a research project to determine the extent to which it was an appropriate model for 
other types of partnerships. 



Q13- To what extent has the ASEP program resulted in skills enhancement 
and work experience opportunities for Aboriginal people that resulted 
in increased ability to participate in the community, compete in the 
labour market and obtain and maintain employment? 

Based on preliminary results, the ASEP projects are demonstrating both tangible and 
intangible outcomes at the individual level. The projects have met their early results 
targets with respect to the number of individuals participating in the project activities. 
Given the data quality issues with the administrative systems, it is challenging to 
determine the extent to which the projects are meeting their early employment target 
results. From the case studies, individuals are reported to have enhanced their skills 
and been provided with work experience opportunities. As the projects progress, the 
challenge faced by some projects will be to assist clients in returning to employment, 
given some of the delays in the timelines for the associated large projects, and sudden 
downturns in some sectors. 



The case studies provided considerable information on both the tangible and intangible 
outcomes to date that clients are experiencing as a result of their participation in the ASEP 
interventions. Many respondents indicated that while it was important to capture and measure 
the more tangible outcomes identified such as the results in the contribution agreements, of 
similar importance was the identification of the less tangible outcomes reported by clients, 
communities, instructors and staff. 

The more tangible outcomes identified by partners, ASEP staff, staff from Aboriginal 
organizations, instructors and clients included the following: 

• Participants moving from dependence on social assistance to economic self-sufficiency; 

• Increased levels of occupation-based skills, essential skills, and life skills; 

• Increased levels of work experience; 

• Increased rates of returning to school and decisions to take further training; 

• Increased income; 

• Increased opportunities to work closer to their communities (for some projects); 

• Increased levels of job search and career planning skills; and 

• Increased opportunities to obtain apprenticeship hours. 

The outcomes that were identified as less tangible, but just as important were: 

• Increased levels of self-confidence and self-esteem; 
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• Increased understanding of the importance of the trades among participants, fa mi lies and 
communities; and 

• Younger generation now observing through participants the benefits of education and 
training. 

While the tangible outcomes are linked in some manner to the ASEP program logic model, the 
intangible outcomes are less evidently linked to the outcomes outlined in the ASEP program 
logic model. 

According to annual reports and performance templates provided by the five case study 
communities, the early stage results targets as established by the contribution agreements had 
been mostly met. This finding contrasts with the finding from the administrative data review. 
As illustrated in Table 4, while the early results targets for number of participants had been 
met by the time the data were extracted (period up to November 2006), the employment rates 
for four of the seven projects that had outcome data available were below the targeted results. 
However, in interpreting this finding, it must be noted that for eight of the nine projects, case 
management systems are not of the quality where the data accurately represents the various 
program activities that are being undertaken by the projects. This summary was extracted from 
the data assessment reports covering the period up to November 2006. Overall, there were 
2,810 clients according the project records - 2,247 clients (80%) were entered into the case 
management databases. Outcomes were recorded for 1,574 clients who showed completed 
action plans. For clients with completed action plans, 408 were coded as employed (26%), 
95 were coded as returned to school (6%) and 848 were coded as unemployed (54%). 
The outcomes were missing or not specified for 234 clients (15%). 



Table 4 

Summary of Number of Clients and Client Outcomes 





Completed Action Plans 8 


First Year Targets 
from Contribution 
Agreements 


Project 


Number of 
Participants 
According to 
Project Records 


Number of 
Participants in 
Case 

Management 

Database 


Number of 
Participants 
with Completed 
Action Plans 


Percentage of 
Employed or 
Self- 

Employed 


Percentage of 
Returned to 
School 


Percentage of 
Unemployed 


Target 
Number of 
Participants 


Target 

Percentage of 
Employed or 
Self-Employed 


VanASEP 
Construction 
Careers Project 


794 


726 


659 


31% 


1% 


68% 


25 


60% 


Aboriginal Mine 
Works 


83 


83 


54 


43% 


7% 


46% 


20 


25% 


Trade Winds to 
Success 


135 


135 


95 


24% 


0% 


41% 


60 


90% 


NWT Industrial 
Mining Skills 
Strategy 


228 


138 


— 


— 


— 


— 


75 


80% 


NWT Oil & Gas 
Skills ASEP 


450 


248 


196 


16% 


6% 


78% 


265 


36% 
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Table 4 (continued) 

Summary of Number of Clients and Client Outcomes 





Completed Action Plans 8 


First Year Targets 
from Contribution 
Agreements 


Project 


Number of 
Participants 
According to 
Project Records 


Number of 
Participants in 
Case 

Management 

Database 


Number of 
Participants 
with Completed 
Action Plans 


Percentage of 
Employed or 
Self- 

Employed 


Percentage of 
Returned to 
School 


Percentage of 
Unemployed 


Target 
Number of 
Participants 


Target 

Percentage of 
Employed or 
Self-Employed 


Manitoba Hydro 
Northern 
Training and 
Employment 
Initiative 


431 


413 


260 


5% 


1% 


22% 


200 


65% 


Victor Diamond 
Mine Project 


234 


155 


155 


19% 


42% 


39% 


255 


10% 


People, Land 
and 

Opportunities 

Project 


264 


178 


155 


54% 


3% 


41% 


125 


44% 


Long Term 
Training 
Initiative for 
Nunavut’s 
Fishing Industry 


191 


171 


— 


— 


— 


— 


40 


50% 


Total 


2,810 


2,247 


1,574 


26% 


6% 


54% 


1,065 


43% 


Source: ASEP project Program Files, Case Management Systems, and Contribution Agreements 
a The row totals may not add to 1 00% due to unspecified outcomes. 



There were several factors influencing the outcomes for the ASEP clients. For the ASEP 
projects studied where there were delays in the timelines associated with the development of 
the industrial projects or a significant downturn in the sector, respondents reported that they 
are experiencing challenges in finding adequate work experience and employment 
opportunities for the clients that have received training and are ready for employment. In these 
communities, respondents report that trained clients are moving out of their communities to 
find employment related to their training. For those who are choosing to remain in their 
communities, they are experiencing challenges in finding employment opportunities related to 
their training and/or the industrial projects. 

With respect to returning to school, one of the main challenges cited within the case studies 
was the occurrence of “social passing”. As a result, some clients are arriving with high school 
credentials but are being assessed at levels far below high school equivalences. There is a large 
gap, then, that exists between their academic level and the level required for them to actively 
participate in post-secondary studies. 



Formative Evaluation of the Aboriginal Skills and Employment Partnership Program 





3.4 Accountability 



Q16- Are adequate safeguards in place to ensure that federal investments 
do not duplicate or displace non-federal investments? 

Q17- Has the ASEP program leveraged additional investments from 
project partners? 

Q18- Are ASEP program investments incremental to existing non-federal 
investments? 



Developing adequate case management and financial systems to ensure that there is no 
duplication or displacement of non-federal investments has been challenging for many 
of the projects, as identified in the data assessments. Although the development of case 
management and financial systems has been challenging, the active involvement of 
partners under a partnership model has assisted in minimizing duplication and overlap 
of investments. The actual measurement of incrementality of investments was not 
possible in this formative evaluation given the lack of baseline investment data prior to 
the implementation of the ASEP program. 



The results from the data assessment indicated that for many projects there will be work 
required to develop their case management and financial systems to adequately ensure 
that there is no duplication or displacement of non-federal investments. For eight of the 
nine projects, case management systems do not accurately represent the various program 
activities that are being undertaken by the projects. 

During the case studies, some project representatives indicated that they were required to 
provide similar levels and types of reporting to both the provincial and federal governments. In 
some cases this was produced by the same case management system, while in other instances, 
this required two separate data systems. 

When asked to comment on the potential for duplication or overlap, some respondents 
indicated that the partnership model, when it works well, is one source of assurance that 
overlap and duplication does not occur. Examples of having partners who are also providing 
funding working in partnership with one another assists in identifying and protecting against 
potential areas of duplication or overlap. 
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4. Key Conclusions 



4.1 Program Rationale and Relevance 

The Aboriginal Skills and Employment Partnership (ASEP) program remains relevant to the 
federal government priorities for Aboriginal human resource development as set out in 
planning and policy documents. Specifically, the design of the ASEP program is consistent 
with the federal priority of building more effective partnerships to improve Aboriginal labour 
market outcomes. The program is also congruent with the recognition that the federal 
government will need to work closely with provincial and territorial partners and Aboriginal 
people, building on established relationships bilaterally and through multilateral fomms. 

The relevance of the ASEP program to the needs of Aboriginal communities is also apparent. 
There is a strong need for Aboriginal people to obtain relevant training and work experience in 
order to more fully participate in the employment opportunities associated with key industrial 
and resource-based projects being developed in specific regions. The demand driven training 
model based on direct input and feedback from industry is an appropriate model to address the 
identified needs. 



4.2 Program Implementation 

While progress has been made in implementing the various intervention activities for the 
ASEP projects, the implementation of case management and financial systems has been 
inadequate. At this stage, the ASEP projects are at risk of not being able to reliably 
demonstrate results at the summative stage given the gaps that currently exist in the case 
management and financial systems. The quality of data provided by these systems also calls 
into question the capacity of the program to adequately perform ongoing monitoring. 
If remedial work on these systems is undertaken right away at the project level, there is a 
strong likelihood that the systems will be able to produce quality data for ongoing monitoring 
and the summative evaluation. 

The attention to providing appropriate services for clients is evident. For most projects, there 
has been considerable effort in ensuring that there is an understanding of the need of the 
clientele, and that project services are adjusted to meet these needs. Clients are generally 
appreciative of this tailoring of the services to meet their needs. The summative evaluation 
should be able to determine the extent to which this effort and success in matching services 
with client needs has resulted in better action plan completion rates, and improved 
employment outcomes for participants. 

The differences were noteworthy in the amount of effort put forward in developing the 
capacity of the governance stmcture and staff. While each project examined had developed 
some rudimentary tools such as by-laws and policies and procedures, some projects placed 
much more emphasis on developing the partnership aspects of the project. These same 
projects were also more likely to invest in the development of the capacity of ASEP staff. 
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and have better communication with other Aboriginal organizations. Where investments in 
time and resources had been made in ensuring the partnership shared a common 
understanding of vision and goals for the ASEP project, there was also a greater l ik elihood 
that the ASEP staff and partners would highlight the partnership and their networks as key 
factors to the success of the organization. In contrast, those organizations that indicated that 
they had invested less effort in developing the partnership were also more likely to highlight 
the partnership as one of the main challenges in achieving the goals and objectives of the 
project. At the summative phase, it will be important to determine the extent to which 
the quality of partnerships and investments in capacity building has had an impact on the 
achievement of project outcomes. 



4.3 Program Results To-Date 

The ASEP program has contributed to an increase in capacity among organizations to address 
the needs of Aboriginal populations with respect to the key issues identified under the mandate 
of the program. In a few cases, the projects were perceived to have limited success in this area 
by providing services that overlap with existing opportunities provided by already established 
programming. 

The ASEP projects evaluated have in most instances met their early targets as outlined in the 
agreements. For some of the projects, the emphasis has been on training and s ki ll development 
for participants in the first few years of the projects. The challenge identified for these projects 
currently and over the next year is to move the trainees into work experiences and support 
them in their first jobs. For the 1,574 clients with completed action plans, 408 were coded 
as employed (26%), 95 were coded as returned to school (6%) and 848 were coded as 
unemployed (54%). The outcomes were missing or not specified for 234 clients (15%). 
Improving on the employment outcomes will be particularly challenging for those projects 
with ties to large industrial projects that have experienced delays or economic downturns 
(e.g., NB forestry sector, Mackenzie pipeline, Manitoba Hydro projects). 



4.4 Accountability 

For eight of the nine projects, case management systems are not of the quality where the data 
accurately represents the various program activities that are being undertaken by the projects. 
This means that data uploaded to HRSDC’s Standard Data File is largely both incomplete and 
inaccurate. This data is used for accountability purposes by Human Resources and S ki lls 
Development Canada (HRSDC). 

This does not suggest that ASEP projects are not operating as intended or they duplicate or 
overlap other activities. In fact, qualitative evidence suggests that the partnership model works 
well to prevent this. However, it does mean that, unless data collection is improved, the 
quantitative data needed to profile clients and activity and demonstrate outcomes and impacts 
will not be available. 
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Appendix A 

ASEP Program Logic Model 



ACTIVITIES 



OUTPUTS 



IMMEDIATE 

OUTCOME 



INTERMEDIATE 

OUTCOME 



ULTIMATE 

OUTCOME 
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